Augmented reality workspace system

ABSTRACT

An augmented reality workspace system includes a physical display device, a user tracking subsystem, and an augmented reality display subsystem, all coupled to a computing system. The computing system uses the augmented reality display subsystem to display an augmented reality element that appears located in a three-dimensional volume immediately adjacent the physical display device as viewed through the augmented reality display subsystem, and uses the physical display device to display a two-dimensional element in a manner that is coordinated with the augmented reality element displayed using the augmented reality display subsystem. When the computing system receives user tracking information from the user tracking subsystem that indicates a user interaction with the augmented reality element, it provides for a coordinated modification of the display of each of the augmented reality element and the two-dimensional element by the augmented reality display subsystem and the physical display device, respectively.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a continuation application to U.S. Utility application Ser. No.15/678,937 filed Aug. 16, 2017, entitled “Augmented Reality WorkspaceSystem,” Attorney Docket No. 16356.1871US01, the disclosure of which isincorporated herein by reference in its entirety.

BACKGROUND

The present disclosure relates generally to information handlingsystems, and more particularly to information handling systems thatprovide augmented reality workspaces.

As the value and use of information continues to increase, individualsand businesses seek additional ways to process and store information.One option available to users is information handling systems. Aninformation handling system generally processes, compiles, stores,and/or communicates information or data for business, personal, or otherpurposes thereby allowing users to take advantage of the value of theinformation. Because technology and information handling needs andrequirements vary between different users or applications, informationhandling systems may also vary regarding what information is handled,how the information is handled, how much information is processed,stored, or communicated, and how quickly and efficiently the informationmay be processed, stored, or communicated. The variations in informationhandling systems allow for information handling systems to be general orconfigured for a specific user or specific use such as financialtransaction processing, airline reservations, enterprise data storage,or global communications. In addition, information handling systems mayinclude a variety of hardware and software components that may beconfigured to process, store, and communicate information and mayinclude one or more computer systems, data storage systems, andnetworking systems.

Information handling systems are sometimes used to provide augmentedreality environments in which a live or indirect view of a physical,real-world environment is augmented by computer-generated sensory inputthat may include video, audio, and/or other data. However, theintegration of augmented reality environments with physical computingworkspaces is limited. In general, conventional physical computingworkspaces that utilize augmented reality environments operate by eitherreplacing physical display devices with the augmented reality workspace(e.g., the users physical environment is augmented with virtual displaydevices that are provided in place of any physical display devices, andthose virtual display devices are then utilized as the workspace), or byproviding the augmented reality elements of the workspace separatelyfrom the physical display devices (e.g., a physical display device isprovided for a user to work on, with augmented reality elements viewableseparately from (e.g., outside the users field of view of) the physicaldisplay devices.) As such, while augmented reality workspaces areavailable, they do no leverage benefits that can be realized when theaugmented reality elements are integrated with physical computingworkspace.

Accordingly, it would be desirable to provide an improved augmentedreality workspace.

SUMMARY

According to one embodiment, an Information Handling System (IHS)includes a processing system; and a memory system is that is coupled tothe processing system and that includes instructions that, when executedby the processing system, cause the processing system to provide anaugmented reality display engine that is configure to: provide, fordisplay on an augmented reality display subsystem and in order to appearto be located in a three-dimensional volume immediately adjacent aphysical display device as viewed through the augmented reality displaysubsystem, a first augmented reality element; provide, for display onthe physical display device, a two-dimensional element that iscoordinated with the first augmented reality element displayed on theaugmented reality display subsystem; receive, from a user trackingsubsystem, first user tracking information; analyze the first usertracking information to determine at least one first user interactionwith the first augmented reality element; modify, in response to the atleast one first user interaction, the display of the first augmentedreality element by the augmented reality display subsystem; and modify,in response to the at least one first user interaction, the display ofthe two-dimensional element by the physical display device in a mannerthat is coordinated with the modification of the display of the firstaugmented reality element by the augmented reality display subsystem.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic view illustrating an embodiment of an informationhandling system.

FIG. 2 is a perspective view illustrating an embodiment of an augmentedreality workspace system.

FIG. 3 is a schematic view illustrating an embodiment of a computingsystem used in the augmented reality workspace system of FIG. 2.

FIG. 4 is a flow chart illustrating an embodiment of a method forproviding an augmented reality workspace.

FIG. 5 is a perspective view illustrating an embodiment of a threedimensional augmented reality volume defined adjacent a physical displaydevice in the augmented reality workspace system of FIG. 2.

FIG. 6A is a perspective view illustrating an embodiment of an augmentedreality element rendered adjacent a physical display device in theaugmented reality workspace system of FIG. 2.

FIG. 6B is a perspective view illustrating an embodiment of atwo-dimensional element displayed on the physical display device in acoordinated manner with the augmented reality element rendered adjacentthe physical display device in the augmented reality workspace system ofFIG. 2.

FIG. 7A is a perspective view illustrating an embodiment of a userinteracting with the augmented reality element of FIG. 6.

FIG. 7B is a perspective view illustrating an embodiment of theaugmented reality element of FIG. 6 coordinated with the display of atwo-dimensional element on the physical display device, and having movedin response to the user interaction of FIG. 7A.

FIG. 7C is a perspective view illustrating an embodiment of theaugmented reality element of FIG. 6 coordinated with the display on thephysical display device, and having moved in response to the userinteraction of FIG. 7A.

FIG. 7D is a perspective view illustrating an embodiment of theaugmented reality element of FIG. 6 coordinated with the display on thephysical display device, and having moved in response to the userinteraction of FIG. 7A.

FIG. 8 is a perspective view illustrating an embodiment of a userinteracting with the augmented reality element of FIG. 6.

FIG. 9 is a perspective view illustrating an embodiment of a userinteracting with the augmented reality element of FIG. 6 in a mannerthat is coordinated with the display on the physical display device.

FIG. 10 is a perspective view illustrating an embodiment of a userinteracting with the augmented reality element of FIG. 6 in a mannerthat is coordinated with the display on the physical display device.

FIG. 11 is a perspective view illustrating an embodiment of theaugmented reality element of FIG. 6 coordinated with the display on thephysical display device.

DETAILED DESCRIPTION

For purposes of this disclosure, an information handling system mayinclude any instrumentality or aggregate of instrumentalities operableto compute, calculate, determine, classify, process, transmit, receive,retrieve, originate, switch, store, display, communicate, manifest,detect, record, reproduce, handle, or utilize any form of information,intelligence, or data for business, scientific, control, or otherpurposes. For example, an information handling system may be a personalcomputer (e.g., desktop or laptop), tablet computer, mobile device(e.g., personal digital assistant (PDA) or smart phone), server (e.g.,blade server or rack server), a network storage device, or any othersuitable device and may vary in size, shape, performance, functionality,and price. The information handling system may include random accessmemory (RAM), one or more processing resources such as a centralprocessing unit (CPU) or hardware or software control logic, ROM, and/orother types of nonvolatile memory. Additional components of theinformation handling system may include one or more disk drives, one ormore network ports for communicating with external devices as well asvarious input and output (I/O) devices, such as a keyboard, a mouse,touchscreen and/or a video display. The information handling system mayalso include one or more buses operable to transmit communicationsbetween the various hardware components.

In one embodiment, IHS 100, FIG. 1, includes a processor 102, which isconnected to a bus 104. Bus 104 serves as a connection between processor102 and other components of IHS 100. An input device 106 is coupled toprocessor 102 to provide input to processor 102. Examples of inputdevices may include keyboards, touchscreens, pointing devices such asmouses, trackballs, and trackpads, and/or a variety of other inputdevices known in the art. Programs and data are stored on a mass storagedevice 108, which is coupled to processor 102. Examples of mass storagedevices may include hard discs, optical disks, magneto-optical discs,solid-state storage devices, and/or a variety other mass storage devicesknown in the art. IHS 100 further includes a display 110, which iscoupled to processor 102 by a video controller 112. A system memory 114is coupled to processor 102 to provide the processor with fast storageto facilitate execution of computer programs by processor 102. Examplesof system memory may include random access memory (RAM) devices such asdynamic RAM (DRAM), synchronous DRAM (SDRAM), solid state memorydevices, and/or a variety of other memory devices known in the art. Inan embodiment, a chassis 116 houses some or all of the components of IHS100. It should be understood that other buses and intermediate circuitscan be deployed between the components described above and processor 102to facilitate interconnection between the components and the processor102.

Referring now to FIG. 2, an embodiment of an augmented reality workspacesystem 200 is illustrated. In the illustrated embodiment, the augmentedreality workspace system 200 includes a computing device 202 that may bethe IHS 100 discussed above with reference to FIG. 1, and/or that mayinclude some of all of the components of the IHS 100. One of skill inthe art in possession of the present disclosure will recognize thatwhile the computing device 202 is illustrated as a desktop computingdevice, other types of computing devices (e.g., laptop/notebookcomputing devices and/or other mobile computing devices, computingdevices integrated into other components of the augmented realityworkspace system 200, and/or other types of computing devices) will fallwithin the scope of the present disclosure as well. As discussed infurther detail below, the computing device 202 may be coupled to othercomponents of the augmented reality workspace system 200 via wiredand/or wireless couplings. Furthermore, while a separate computingdevice 202 is illustrated in FIG. 2, the functionality of the computingdevice 202 may instead by provided by a computing system that may be atleast partially distributed across the components of the augmentedreality workspace system 200.

For example, the augmented reality workspace system 200 of theillustrated embodiment includes a physical display device 204 that isconnected to the computing device 202 by a wired connection 206,although wireless connections between the computing device 202 and thephysical display device 204 (or integration of at least some of thecomputing device functionality discussed below in the physical displaydevice 204) will fall within the scope of the present disclosure aswell. In an embodiment, the physical display device 204 may include thedisplay 110 discussed above with reference to FIG. 1. The physicaldisplay device 204 includes a display screen 204 a that, in theembodiments illustrated and discussed below, is provided in asubstantially horizontal orientation relative to a user of the augmentedreality workspace system 200, as well as substantially parallel to thesupport surface upon which it is located (e.g., a working surface of adesk.) For example, one of skill in the art in possession of the presentdisclosure will recognize that display screens have been traditionallyprovided in substantially vertical orientations relative to users, aswell as substantially perpendicularly to their support surfaces (e.g.,the working surface of the desk discussed above), and that the displayscreen 204 a of the physical display device 204 is described below asprovided in a substantially horizontal orientation that is rotatedsubstantially ninety degrees from those substantially verticalorientations. For example, the physical display device 204 may beprovided as part of a “smart desk” that provides a horizontallyoriented, touch-input display device (which may be utilized by itself orin conjunction with a vertically oriented display device), althoughother horizontally oriented display screens will fall within the scopeof the present disclosure as well. Furthermore, the provisioning of thephysical display device 204 and its display screen 204 a in otherorientations (e.g., the vertical orientation discussed above) will fallwithin the scope of the present disclosure as well.

In the illustrated embodiment, a user tracking subsystem 208 a and 208 bis integrated with the physical display device 204, although a usertracking subsystem that is separate from the physical display device 204(and separately coupled to the computing device 202 via a wired orwireless connection) will fall within the scope of the presentdisclosure as well. As such, in some embodiments the user trackingsubsystem 208 and 208 b may include at least some of the computingdevice functionality described below for the physical display device204. The user tracking subsystem 208 a and 208 b may include a pluralityof user tracking devices 208 c that may be provided by infrared (IR)sensors, IR sensor arrays (e.g., “IR castors”), three-dimensionalcameras (e.g., if the processing system in the computing system hassufficient processing capabilities), and/or a variety of other usertracking devices that would be apparent to one of skill in the art inpossession of the present disclosure. While the augmented realityworkspace system 200 is illustrated with the user tracking subsystem 208a positioned at the “top” of the physical display device 204 and theuser tracking subsystem 208 b positioned at the “bottom” of the physicaldisplay device 204, user tracking subsystems with different numbers ofcomponents in different configurations and/or orientations will fallwithin the scope of the present disclosure as well.

In the illustrated embodiment, an augmented reality display subsystem210 is included with the augmented reality workspace system 200, andprovides a head-mounted user tracking and display subsystem. Forexample, the augmented reality display subsystem 210 includes a chassis210 a that is configured to be worn on a user's head such that a displaydevice 210 b is positioned in front of the user's eyes. In thediscussions below, the display device 210 b is provided by a transparentOrganic Light Emitting Device (OLED) display device, although otherdisplay devices that provide the functionality discussed below may fallwithin the scope of the present disclosure as well. The augmentedreality display subsystem 210 may also include a plurality of cameras210 c that are configured to capture images in the field of view of auser wearing the augmented reality display subsystem 210. In theexamples discussed below, the augmented reality display subsystem 210 iswirelessly coupled to the computing device 202, although wiredconnections will fall within the scope of the present disclosure aswell. While in the embodiments discussed below, much of the computingdevice processing for the display of images by the augmented realitydisplay subsystem 210 is performed by the computing device 202 in orderto provide a relatively small and lightweight augmented reality displaysubsystem 210, in other embodiments the augmented reality displaysubsystem 210 may perform at least some of the computing devicefunctionality discussed below. While not explicitly illustrated, theaugmented reality display subsystem 210 may include a variety of othercomponents for use in the user tracking functionality discussed below,including IR markers (e.g., for use by IR sensors or IR sensor arrays inthe user tracking subsystem 208 a and 208 b), accelerometers,gyroscopes, locations sensors, and/or a variety of other trackingcomponents that would be apparent to one of skill in the art inpossession of the present disclosure. In experimental embodiments, theaugmented reality display subsystem 210 was provided by a META 2®headset provided by META® company of California, United States, althoughother augmented reality display subsystems will fall within the scope ofthe present disclosure as well. However, while a specific augmentedreality display subsystem has been described, one of skill in the art inpossession of the present disclosure will recognize that light fielddisplay devices, projection display devices, and/or other augmentedreality display subsystems may be substituted for the augmented realitydisplay subsystem 210 while remaining within the scope of the presentdisclosure.

In the illustrated embodiment, the augmented reality workspace system200 also includes a totem device 212 and a pen device 214, each of whichmay be wirelessly connected to the computing device 202 (although wiredconnections will fall within the scope of the present disclosure aswell), or capable of being tracked by the augmented reality displaysubsystem 210 and/or the user tracking subsystem 208 a and 208 b.Furthermore, each of the totem device 212 and the pen device 214 mayinclude tracking components such as IR markers (e.g., for use by IRsensors or IR sensor arrays in the user tracking subsystem 208 a and 208b), cameras, accelerometers, gyroscopes, locations sensors, and/or avariety of other tracking components that would be apparent to one ofskill in the art in possession of the present disclosure. While aspecific augmented reality workspace system has been described, one ofskill in the art in possession of the present disclosure will recognizethat augmented reality workspace systems may include a variety ofcomponents in a variety of different configurations in order to providefor conventional augmented reality workspace functionality, as well asthe functionality discussed below, while remaining within the scope ofthe present disclosure.

Referring now to FIG. 3, an embodiment of a computing device 300 isillustrated that may be the computing device 202 discussed above withreference to FIG. 2. As such, the computing device 300 may be the IHS100 discussed above with reference to FIG. 1, and/or may include some orall of the components of the IHS 100, and in specific embodiments may bea desktop computing device (although other types of computing deviceswill fall within the scope of the present disclosure as well, asdiscussed above.) Furthermore, as discussed above, while a separatecomputing device 300 is illustrated in FIG. 300, the functionality ofthe computing device 300 may instead by provided by a computing systemthat may be at least partially distributed across the components of theaugmented reality workspace system 200. In the illustrated embodiment,the computing device 300 includes a chassis 302 that houses thecomponents of the computing device 300, only some of which areillustrated in FIG. 3. For example, the chassis 302 may house aprocessing system (not illustrated, but which may include the processor102 discussed above with reference to FIG. 1) and a memory system (notillustrated, but which may include the system memory 114 discussed abovewith reference to FIG. 1) that is coupled to the processing system andthat includes instructions that, when executed by the processing system,cause the processing system to provide an augmented reality displayengine 304 that is configured to perform the functions of the augmentedreality display engines and computing devices discussed below. However,as discussed above, rather than being provided in a separate computingdevice, the functionality and/or processing performed by the computingdevice as discussed below may instead be integrated into components ofthe augmented reality workspace system 200 (e.g., the physical displaydevice 204, the user tracking subsystem 208 a and 208 b, the augmentedreality display subsystem 210, etc.) while remaining within the scope ofthe present disclosure.

In the illustrated embodiment, the augmented reality display engine 304includes a user tracking sub-engine 304 a that may be configured toutilize user tracking information to determine the position of the user(e.g., the user's head, the user's hands, and/or other portions of theuser), a two-dimensional visualization sub-engine 304 b that may beconfigured to generate the two-dimensional elements on the displayscreen 204 a of the physical display device 204, a three-dimensionalvisualization sub-engine 304 c that may be configured to generate theaugmented reality elements via the augmented reality display subsystem210, and a color sub-engine 304 c that may be configured to determinecolor details of the two-dimensional and augmented reality elementsgenerates by the two-dimensional visualization sub-engine 304 b and thethree-dimensional visualization sub-engine 304 c. However, while anexample of specific sub-engines and components of the augmented realitydisplay engine 304 have been illustrated and are described in moredetail below, one of skill in the art in possession of the presentdisclosure will recognize that the augmented reality display engine 304may include more or fewer sub-engines, and those sub-engines may bedistributed across multiple different components of the augmentedreality workspace system 200 (e.g., the user tracking sub-engine 304 aprovided in the user tracking subsystem 208 a and 208 b, two-dimensionalvisualization sub-engine 304 b provided in the physical display device204, the three-dimensional visualization sub-engine 304 c provided inthe augmented reality display subsystem 210, etc.) while remainingwithin the scope of the present disclosure.

The chassis 302 may also house a storage system (not illustrated, butwhich may include the storage device 108 discussed above with referenceto FIG. 1) that is coupled to the augmented reality display subsystem304 (e.g., via a coupling between the storage system and the processingsystem) and that may include an augmented reality display database 306that is configured to store any of the information that is used toprovide the functionality discussed below. The chassis 302 may alsohouse a communication subsystem 308 that is coupled to the augmentedreality display subsystem 304 (e.g., via a coupling between thecommunication subsystem 308 and the processing system) and that mayinclude a Network Interface Controller (NIC) (e.g., for providing thewired connections discussed above), a wireless communication device(e.g., a BLUETOOTH® communications device, a Near Field Communications(NFC) device, a WiFi communications devices, and/or other wirelesscommunications devices for providing the wireless connections discussedabove), and/or other communications components that would be apparent toone of skill in the art in possession of the present disclosure. While aspecific computing device has been illustrated and described, one ofskill in the art in possession of the present disclosure will recognizethat computing devices may include a variety of components in a varietyof configurations in order to provide conventional computing devicefunctionality, as well as the functionality discussed below, whileremaining within the scope of the present disclosure.

Referring now to FIG. 4, an embodiment of a method 400 for providing anaugmented reality workspace is illustrated. As discussed below, at leastsome embodiments of the systems and methods of the present disclosureprovide for the integration of an augmented reality workspace with oneor more physical display devices in order to provide an immersiveworkspace experience that coordinates the display of augmented realityelements via an augmented reality display subsystem with the display oftwo-dimensional elements via a physical display device that provides forproductivity increases using the system. For example, the design ofproducts using three-dimensional augmented reality elements displayedvia the augmented reality display subsystem is enhanced by providing forthe display of “projections”, cross-sections, surfaces, and/or otherfeatures of the three-dimensional element onto the physical displaydevice using the two-dimensional element displayed thereon, allowing forpreviously unavailable visualizations of the product being designed,engagement surfaces on the product being designed, and/or a variety ofother benefits that will be apparent to one of skill in the art inpossession of the present disclosure. Furthermore, the system may beenhanced using peripheral devices such a totems, which allow formanipulation of the three-dimensional element displayed via theaugmented reality display subsystem, and pens, which may be utilizedwith the physical display device to allow for the augmentation ofthree-dimensional element displayed via the augmented reality displaysubsystem.

The method 400 begins at block 402 where a computing system uses anaugmented reality display subsystem to display an augmented realityelement in a three dimensional volume adjacent a physical displaydevice. In the embodiments discussed below, for clarity of illustration,the augmented reality workspace system 200 is illustrated with only thephysical display device 204 and without the augmented reality displaysubsystem 210 and the user tracking subsystem 208 a and 208 b. However,one of skill in the art in possession of the present disclosure willrecognize that the augmented reality display subsystem 210 and the usertracking subsystem 208 a and 208 b may be utilized throughout the method400 to provide the functionality discussed below. With reference to FIG.5, at or before block 402 (e.g., upon and/or during startup of theaugmented reality workspace system 200, the user tracking sub-engine 304a and the three-dimensional visualization sub-engine 304 c in theaugmented reality display engine 304 may operate to receive informationprovided by the augmented reality display subsystem 210 and the usertracking subsystem 208 a and 208 b, and define a three-dimensionalvolume 500 immediately adjacent the display screen 204 a on the physicaldisplay device 204 a.

For example, the user tracking sub-engine 304 a in the augmented realitydisplay engine 304 may receive user tracking information that may begenerated by the augmented reality display subsystem 210 (e.g., usertracking information related to the location, orientation, and/ormovement of the user's head), generated by the user tracking subsystem208 a and 208 b (e.g., user tracking information related to thelocation, orientation, and/or movement of any part of the user's bodydetectable by the user tracking subsystem 208 a and 208 b), and/orgenerated by any other component in the system 200. In addition, thethree-dimensional visualization sub-engine 304 c in the augmentedreality display engine 304 may retrieve physical display deviceinformation (e.g., information describing the location, orientation,size, and/or other characteristics of the physical display device 204)from the physical display device 204 and/or from the augmented realitydisplay database 306, as well as receive image information from thecameras 210 c on the augmented reality display subsystem 210. The usertracking sub-engine 304 a may then provide the user tracking informationto the three-dimensional visualization sub-engine 304 c, and thethree-dimensional visualization sub-engine 304 c may use the usertracking information, the physical display device information, the imageinformation, and/or any other available information to recognize thephysical display device 204 in real-time as it is being viewed throughthe augmented reality display subsystem 210, and define thethree-dimensional volume 500 adjacent the display screen 204 a on thephysical display device 204 as it is viewed through the augmentedreality display subsystem 210. As such, one of skill in the art inpossession of the present disclosure will recognize that the threedimensional volume 500 remains stationary relative to the view of thephysical display device 204 via the augmented reality display subsystem210 even when a user wearing that augmented reality display subsystem210 moves their head and changes their view of the physical displaydevice 204.

While a specific example of the definition of the three-dimensionalvolume adjacent the physical display device 204 has been described, oneof skill in the art in possession of the present disclosure willrecognize that a variety of techniques for defining such athree-dimensional volume adjacent a physical object will fall within thescope of the present disclosure as well. Furthermore, while thethree-dimensional volume 500 is illustrated and has been described asbeing defined by the boundaries of the display screen 204 a on thephysical display device 204, the three-dimensional volume 500 may bedefined to extend outside the boundaries of the display screen 204 a, orwithin the boundaries of the display screen 204 a, while remainingwithin the scope of the present disclosure. Further still, while thethree-dimensional volume 500 is illustrated as having an upper boundary,in some embodiments, the three-dimensional volume may extend upwardwithout a boundary.

In some embodiments, the definition of the three dimensional volume 500adjacent the physical display device 204 may create a plurality ofdisplays zones. For example, a primary display zone may exist within thethree dimensional volume 500 and may be utilized to provide for thedisplay of augmented reality elements that are coordinated with thedisplay of two-dimensional elements on the physical display device 204as discussed below, a secondary zone may exist immediately adjacent thethree-dimensional volume 500 and may be utilized to provide for thedisplay of augmented reality elements such as tools, menus, andlibraries, and a tertiary zone may exist outside of the secondary zoneand may be utilized to provide for the display of augmented realityelements that provide reference materials, content consumption, and/orother information. However, while specific display zones have beendescribed, one of skill in the art in possession of the presentdisclosure will recognize that display zones may be defined for avariety of different purposes other than those mentioned above whileremaining within the scope of the present disclosure.

Referring now to FIG. 6A, in an embodiment of block 402, thethree-dimensional visualization sub-engine 304 c in the augmentedreality display engine 304 may operate to use the augmented realitydisplay subsystem 210 to render, display, or otherwise provide anaugmented reality element 600 in the three dimensional volume 500adjacent the physical display device 204. For example, at block 402, auser of the system 200 may provide the three-dimensional visualizationsub-engine 304 c object information that describes how the augmentedreality element 600 should be displayed, and/or instruct thethree-dimensional visualization sub-engine 304 c to retrieve objectinformation from the augmented reality display database 306 thatdescribes how the augmented reality element 600 should be displayed.

At block 402, the user tracking sub-engine 304 a in the augmentedreality display engine 304 may also receive user tracking informationthat may be generated by the augmented reality display subsystem 210(e.g., user tracking information related to the location, orientation,and/or movement of the user's head), generated by the user trackingsubsystem 208 a and 208 b (e.g., user tracking information related tothe location, orientation, and/or movement of any part of the user'sbody detectable by the user tracking subsystem 208 a and 208 b), and/orgenerated by any other component in the system 200. Furthermore, atblock 402 the three-dimensional visualization sub-engine 304 c in theaugmented reality display engine 304 may receive image information fromthe cameras 210 c on the augmented reality display subsystem 210. Theuser tracking sub-engine 304 a may then provide that user trackinginformation to the three-dimensional visualization sub-engine 304 c, andthe three-dimensional visualization sub-engine 304 c may use the usertracking information, the physical display device information, the imageinformation, the object information, and/or any other availableinformation to generate augmented reality element information andprovide that augmented reality element information to the augmentedreality display subsystem 210. The augmented reality display subsystem210 may then use that augmented reality element information to displaythe augmented reality element 600 in the three-dimensional volume 500adjacent the display screen 204 a on the physical display device 204 asit is viewed through the augmented reality display subsystem 210. Assuch, one of skill in the art in possession of the present disclosurewill recognize that, absent a user interaction with the augmentedreality element 600 (discussed below), the augmented reality element 600may remain stationary relative to the view of the physical displaydevice 204 via the augmented reality display subsystem 210 even when auser wearing that augmented reality display subsystem 210 moves theirhead and changes their view of the physical display device 204.

In an embodiment of block 402, the color sub-engine 304 d in theaugmented reality display engine 304 may operate to adjust colors thatare used to display the augmented reality element 600 to provide for acoloring on the augmented reality element 600 that is consistent and/orcoordinated with the coloring of the physical space in which it is beingviewed. For example, the color sub-engine 304 d may utilize gamut andhue feedback from the cameras 210 c and/or other sensors on theaugmented reality display subsystem 210 (and/or other components in thesystem 200), receive information from ambient light sensors provided inthe augmented reality display subsystem 210 (and/or other components inthe system 200), utilize color information, brightness information,and/or other display information from the physical display device 204,and compare difference in hue and gamut, information related to ambientlight, and the physical display device information discussed above inorder to render the color utilized on the augmented reality element 600at block 402 (and in later blocks of the method 400.)

While a specific example of the display of an augmented reality elementadjacent the physical display device 204 has been described, one ofskill in the art in possession of the present disclosure will recognizethat a variety of techniques for rendering, displaying, or otherwiseproviding such an augmented reality element adjacent a physical objectwill fall within the scope of the present disclosure as well.Furthermore, while the augmented reality element 600 is illustrated asbeing a three-dimensional element, the augmented reality element 600 maybe displayed as a two-dimensional element while remaining within thescope of the present disclosure as well.

The method 400 then proceeds to block 404 where the computing systemuses the physical display device to display a two dimensional elementthat is coordinated with the augmented reality element. Referring now toFIG. 6B, in an embodiment of block 404, the two-dimensionalvisualization sub-engine 304 b in the augmented reality display engine304 may operate in cooperation with the three-dimensional visualizationsub-engine 304 c in the augmented reality display engine 304 to render,display, or otherwise provide a two-dimensional element 602 on thedisplay screen 204 a of the physical display device 204, with thattwo-dimensional element coordinated with the three-dimensional element600 displayed adjacent the physical display device 204. For example, atblock 404, a user of the system 200 may have provided thetwo-dimensional visualization sub-engine 304 b object information thatdescribes how the two-dimensional element 602 should be displayed,and/or instruct the two-dimensional visualization sub-engine 304 b toretrieve object information from the augmented reality display database306 that describes how the two-dimensional element 602 should bedisplayed.

For example, at block 404, the two-dimensional visualization sub-engine304 b in the augmented reality display engine 304 may use the objectinformation, as well as the augmented reality element informationgenerated by the three-dimensional visualization sub-engine 304 c, togenerate two-dimensional element information and provide thattwo-dimensional element information to the physical display device 204.The physical display device 204 may then use that two-dimensionalelement information to display the two-dimensional element 602 on thedisplay screen 204 a of the physical display device 204 in a coordinatedmanner with the augmented reality element 600. In some embodiments, thetwo-dimensional element 602 may be coordinated with the augmentedreality element 600 by providing a cross-sectional view of the augmentedreality element 600. For example, the two-dimensional element 602 mayprovide a cross-sectional view of the augmented reality element 600along the dotted line illustrated in the augmented reality element 600of FIG. 6B (which may be selected by the user via the user interactionsdiscussed below.) In some embodiments, the two-dimensional element 602may be coordinated with the augmented reality element 600 by providing aprojected view of the augmented reality element 600. For example, thetwo-dimensional element 602 may provide a projected view of theaugmented reality element 600 that illustrates a surface of theaugmented reality element 600, that illustrates a shadow cast by theaugmented reality element (e.g., in response to an augmented realitylight source that may be positioned as desired by the user), and/or thatillustrated a variety of other projected views that would be apparent toone of skill in the art in possession of the present disclosure.

In some embodiments, the two-dimensional element 602 may be coordinatedwith the augmented reality element 600 by providing a surface-engagementview of the augmented reality element 600. For example, augmentedreality element 600 may include a textured surface (not illustrated),and the two-dimensional element 602 may provide a surface-engagementview of the augmented reality element 600 that illustrates how thetextured surface of the augmented reality element 600 would engage asupport surface, the amount of force that would be experienced bydifferent portions of the textured surface of the augmented realityelement 600 when engaging a support surface, and/or a variety of othersurface-engagement views that would be apparent to one of skill in theart in possession of the present disclosure. While specific examples ofthe display of two-dimensional elements coordinated with augmentedreality elements have been described, one of skill in the art inpossession of the present disclosure will recognize that a variety oftechniques for rendering such coordinated two-dimensionalelements/augmented reality elements will fall within the scope of thepresent disclosure as well.

As discussed in detail below, following block 404 the computing device300 operates to detect user interactions at block 406, and then modifythe display of the augmented reality element 600 at block 408, andmodify the display of two-dimensional element 602 at block 410 in amanner that is coordinated with the augmented reality element 600.Several different examples of user interactions and the associatedcoordinated modifications of the display of the augmented realityelement 600 and the two-dimensional element 602 are described separatelybelow, but one of skill in the art in possession of the presentdisclosure will recognize that any of functionality described in theexamples below may be combined with functionality described in the otherexamples while remaining within the scope of the present disclosure.Furthermore, the specific examples provided below are not meant to belimiting, and one of skill in the art in possession of the presentdisclosure will recognize how the teachings provided herein may beextended to a variety of functionality that will fall within the scopeof the present disclosure.

In a first embodiment, the method 400 may proceed to block 406 where thecomputing system detects a user interaction with the augmented realityelement. In the first embodiment, at block 406, the user trackingsub-engine 304 a in the augmented reality display engine 304 may receiveuser tracking information that may be generated by the user trackingsubsystem 208 a and 208 b (e.g., user tracking information related tothe location, orientation, and/or movement of any part of the user'sbody detectable by the user tracking subsystem 208 a and 208 b), and/orgenerated by any other component in the system 200. The user trackingsub-engine 304 a may then provide that user tracking information to thethree-dimensional visualization sub-engine 304 c. Furthermore, at block406 the three-dimensional visualization sub-engine 304 c in theaugmented reality display engine 304 may receive image information fromthe cameras 210 c on the augmented reality display subsystem 210. Atblock 406, the three-dimensional visualization sub-engine 304 c may thenuse the user tracking information, the image information, and/or anyother available information to detect at least one user interaction withthe augmented reality element 600.

Referring now to FIG. 7A, the rendering, displaying, or otherwiseprovisioning of the augmented reality element 600 by thethree-dimensional visualization sub-engine 304 c in the augmentedreality display engine 304 may include the defining of a plurality ofaugmented reality element axis, including the augmented reality elementX-axis 700 a, the augmented reality element Y-axis, and the augmentedreality element Z-axis 700 c illustrated in FIG. 7A. In this firstembodiment of block 406, the user may move their hand 702 into thethree-dimensional volume 500 located adjacent the physical displaydevice 204 and perform a hand gesture A that, in FIG. 7A, is illustratedas a “pinch” gesture, and that is followed by a translational movementof the hand 702. The user tracking sub-engine 304 a may receive the usertracking information generated by the user tracking subsystem 208 a and208 b in response to that hand movement, and then provide that usertracking information to the three-dimensional visualization sub-engine304 c. The three-dimensional visualization sub-engine 304 c may alsoreceive image information generated by the camera(s) 210 c on theaugmented reality display subsystem 210 in response to that handmovement as well. The three-dimensional visualization sub-engine 304 cmay then analyze the user tracking information, the image information,and/or any other available information to detect the hand gesture Abeing performed within the three-dimensional volume 500, followed by thetranslational movement of the hand 702. In the specific embodimentillustrated in FIG. 7A, the hand gesture A is detected adjacent theaugmented reality element Z-axis 700 c prior to the translationalmovement of the hand 702, and at block 406 that hand gesture A andaugmented reality element Z-axis 700 c may be associated with eachother.

In the examples illustrated and described below, the hand gesture A(i.e., a pinch gesture) located adjacent an augmented reality elementaxis is associated with movement of the augmented reality element 600.However, as would be understood by one of skill in the art in possessionof the present disclosure, a variety of different hand gestures may beassociated with a variety of different augmented reality element actionswhile remaining within the scope of the present disclosure. For example,a pinch gesture that is not located adjacent an augmented realityelement axis may be associated with a “zoom” action that causes theaugmented reality element 600 to be displayed smaller, and areverse-pinch gesture (i.e., when the fingers on a user's hand are movedapart from each other) may be associated with an “unzoom” action thatcauses the augmented reality element 600 to be displayed larger. Inanother example, a two-finger gesture (i.e., when an index and middlefinger on a user's hand are held out while the remaining fingers arenot) may be associated with movement of the augmented reality element600. However, while a plurality of different hand gestures have beendescribed, one of skill in the art in possession of the presentdisclosure will recognize that any of a variety of hand gestures may beassociated with any of a variety of augmented reality element actionswhile remaining within the scope of the present disclosure.

Furthermore, as also illustrated in FIG. 7A, a user interaction may bedetected via totem device information generated in response totranslational movement of the totem device 212. For example, the usermay use their hand 702 to move the totem device 212 along a totem deviceX-axis 702 a or a totem device Y-axis 702 b, and that translationalmovement may be reported to the three-dimensional visualizationsub-engine 304 c as totem device information (e.g., reported directly bythe totem device 212, detected and reported by the user trackingsub-engine 304 a, etc.) As such, at block 406, the three-dimensionalvisualization sub-engine 304 c may use the totem device informationgenerated via translational movement of the totem device 212 to detectthe at least one user interaction with the augmented reality element600. In some embodiments, the totem device 212 may be utilized on thesurface of the display screen 204 a of the physical display device 204.In such embodiments, the two-dimensional visualization engine 304 b maybe configured to recognize when a user's palm is engaging the displayscreen 204 a along with the totem device 212, and operate to disregardthat engagement of the user's palm.

In the first embodiment, the method 400 may then proceed to block 408where the computing system uses the augmented reality display subsystemto modify the display of the augmented reality element in the threedimensional volume adjacent the physical display device based on theuser interaction. With reference to FIGS. 7A, 7B, 7C, and 7D, in anembodiment of block 408, the three-dimensional visualization sub-engine304 c may determine that the user interaction detected at block 406 and,in response, modify the display of the augmented reality element 600 inthe three dimensional volume 500. In this example, the hand gesture A isa pinch gesture that is performed adjacent the augmented reality elementZ-axis 700 c, and is followed by translational movement of the hand 702along the augmented reality element Z-axis 700 c and towards thephysical display device 204. Based on that detected user interactionwith the augmented reality element 600, the augmented realityvisualization sub-engine 304 c may modify the display of the augmentedreality element 600 provided by the augmented reality display subsystem210 to cause the augmented reality element 600 to appear to perform atranslational movement through the three-dimensional volume 500 andtowards the physical display device 204, as illustrated in FIGS. 7A, 7B,7C, and 7D.

In the embodiment illustrated in FIGS. 7B, 7C, and 7D, as the usercontinues to perform the translational movement of their hand 702 whileperforming the hand gesture A adjacent the augmented reality elementZ-axis 700 c, the modified display of the augmented reality element 600will cause the augmented reality element 600 to reach the intersectionof the three-dimensional volume 500 and the display screen 204 a on thephysical display device 204, and appear to move into contact with thedisplay screen on the physical display device 204. In response to thisuser interaction continuing (e.g., the continued translational movementof the hand 702 while performing the hand gesture A adjacent theaugmented reality element Z-axis 700 c), the augmented reality element600 will move past the intersection of the three-dimensional volume 500and the display screen 204 a on the physical display device 204. Asillustrated in FIGS. 7C and 7D, movement of the augmented realityelement 600 past the intersection of the three-dimensional volume 500and the display screen 204 a on the physical display device 204 maycause the augmented reality visualization sub-engine 304 c to modify thedisplay of the augmented reality element 600 provided by the augmentedreality display subsystem 210 to cause any portion of the augmentedreality element 600 that moves past the intersection of thethree-dimensional volume 500 and the display screen 204 a on thephysical display device 204 to not be displayed (i.e., the augmentedreality element 600 appears to “disappear” into the display screen 204a.)

However, while the specific example in FIGS. 7B, 7C, and 7D illustratesa vertical translational movement of the augmented reality element 600,one of skill in the art in possession of the present disclosure willrecognize that horizontal translational movements of the augmentedreality element 600 will fall within the scope of the present disclosureas well. For example, the hand gesture A may be a pinch gesture that isperformed adjacent the augmented reality element X-axis 700 b or Y-axis700 c, and may be followed by translational movement of the hand 702along that augmented reality element X-axis 700 a or Y-axis 700 b. Inresponse, the augmented reality visualization sub-engine 304 c may causethe display of the augmented reality element 600 to appear to move alongthat augmented reality element X-axis 700 b or Y-axis 700 c insubstantially the same manner as discussed above. Similarly as discussedabove, movement of the augmented reality element 600 in the direction ofthe augmented reality element X-axis 700 b or Y-axis 700 c may cause theaugmented reality element 600 to reach a boundary of thethree-dimensional volume 500, and may result in portions of theaugmented reality element 600 that move beyond that boundary to“disappear” or provide for the viewing of a cross section of theaugmented reality element 600 (i.e., the cross section of the augmentedreality element 600 immediately adjacent that boundary.) However,continued display of portions of the augmented reality element 600 thatmove beyond such a boundary on the three-dimensional volume 500 willfall within the scope of the present disclosure as well.

Furthermore, the hand gesture A performed adjacent any of the augmentedreality element axis as discussed above may allow the user to translatetheir hand 702 in the direction of any of those augmented realityelement axis to have the display of the augmented reality element 600appear to translate in substantially the same manner as discussed above.Further still, the movement of the totem device 212 as discussed above(e.g., along the totem device X-axis 702 a or the totem device Y-axis702 b) may cause the augmented reality element 600 to move along acorresponding augmented reality element axis (e.g., the augmentedreality element X-axis 700 b or Y-axis 700 c) in substantially the samemanner as discussed above. In some embodiments, the totem device 212 maybe utilized on the surface of the display screen 204 a of the physicaldisplay device 204. In such embodiments, the two-dimensionalvisualization engine 304 b may be configured to recognize when a user'spalm is engaging the display screen 204 a along with the totem device212, and operate to disregard that engagement of the user's palm. Assuch, the modification of the display of the augmented reality element600 may be caused by a variety of user actions while remaining withinthe scope of the present disclosure.

In the first embodiment, the method 400 may then proceed to block 410where the computing system uses the physical display device to modifythe display of the two dimensional element that is coordinated with theaugmented reality element. With reference to FIGS. 7A and 7B, thetranslational movement of the augmented reality element 600 may causethe two-dimensional visualization sub-engine 304 b to modify the displayof the two-dimensional element 602 to provide a modified two-dimensionalelement 704. For example, as discussed above, the two-dimensionalelement 602 may be coordinated with the augmented reality element 600 byproviding a cross-sectional view of the augmented reality element 600,and in this example that cross-sectional view may be defined by ahorizontally oriented static plane (e.g., a plane that includes theaugmented reality element X-axis 700 a and the augmented reality elementY-axis 700 b in FIG. 7A, and that is parallel to the display screen 204a) that intersects the augmented reality element 600. In response to thetranslational movement of the augmented reality element 602, therelative position of the augmented reality element 602 and that staticplane may change, and result in the coordinated display of the modifiedtwo-dimensional element 704 that provides a different cross-section ofthe augmented reality element 602 (e.g., as indicated by the dottedcircle 706 in FIG. 7B.) As such, different cross-sections of theaugmented reality element 600 may be viewed on the physical displaydevice 304 by performing user interactions that translate the augmentedreality element 600 as discussed above. One of skill in the art inpossession of the present disclosure will recognize how across-sectional view of the augmented reality element 600 may beprovided with a vertical static plane (i.e., a static plane thatincludes the augmented reality element Y-axis 700 b and the augmentedreality element Z-axis 700 c in FIG. 7A, and that is perpendicular tothe display screen 204 a) in substantially the same manner as describedabove.

As discussed above, the embodiments illustrated in FIGS. 7C and 7Ddescribe an example where the movement of the augmented reality element600 past the intersection of the three-dimensional volume 500 and thedisplay screen 204 a on the physical display device 204 causes theaugmented reality visualization sub-engine 304 c to modify the displayof the augmented reality element 600 provided by the augmented realitydisplay subsystem 210 to cause any portion of the augmented realityelement 600 that moves past the intersection of the three-dimensionalvolume 500 and the display screen 204 a on the physical display device204 to not be displayed (i.e., to cause the augmented reality element600 to appear to “disappear” into the display screen 204 a.) In suchembodiments, the cross-sectional view of (or other two-dimensionalelement coordination with) the augmented reality element 600 may bevisualized in greater detail when the augmented reality element 600intersects the three-dimensional volume 500 and the display screen 204 aon the physical display device 204.

In other examples, the coordinated modification of the two-dimensionalelement 602 with the augmented reality element 600 may include providinga modified projected view of the augmented reality element 600. Forexample, the two-dimensional element 704 may provide a projected view ofthe augmented reality element 600 that illustrates different portions ofthe surface of the augmented reality element 600 as it translates(possible along the augmented reality element X-axis 700 a or theaugmented reality element Y-axis 700 c rather than the augmented realityelement Z-axis 700 c in the illustrated embodiment), a projected viewthat illustrates a shadow cast by the augmented reality element 602 asit translates (e.g., in response to the augmented reality light sourcethat may be positioned as desired by the user), and/or that illustrateda variety of other projected views that would be apparent to one ofskill in the art in possession of the present disclosure.

In yet other examples, the coordinated modification of thetwo-dimensional element 602 with the augmented reality element 600 mayinclude a modified surface-engagement view of the augmented realityelement 600. As discussed above, augmented reality element 600 mayinclude a textured surface (not illustrated), and the two-dimensionalelement 704 may provide a surface-engagement view of the augmentedreality element 600 that illustrates how the textured surface of theaugmented reality element 602 would engage a support surface as theaugmented reality element 602 translates, the amount of force that wouldbe experienced by different portions of the textured surface of theaugmented reality element 602 as it translates when engaging a supportsurface, and/or a variety of other surface-engagement views that wouldbe apparent to one of skill in the art in possession of the presentdisclosure. While specific examples of the display of modifiedcoordination of the display of two-dimensional elements with augmentedreality elements have been described, one of skill in the art inpossession of the present disclosure will recognize that a variety oftechniques for rendering such modified coordinated two-dimensionalelements/augmented reality elements will fall within the scope of thepresent disclosure as well.

In a second embodiment, the method 400 may proceed to block 406 wherethe computing system detects a user interaction with the augmentedreality element. In the second embodiment, at block 406, the usertracking sub-engine 304 a in the augmented reality display engine 304may receive user tracking information that may be generated by the usertracking subsystem 208 a and 208 b (e.g., user tracking informationrelated to the location, orientation, and/or movement of any part of theuser's body detectable by the user tracking subsystem 208 a and 208 b),and/or generated by any other component in the system 200. The usertracking sub-engine 304 a may then provide that user trackinginformation to the three-dimensional visualization sub-engine 304 c.Furthermore, at block 406 the three-dimensional visualization sub-engine304 c in the augmented reality display engine 304 may receive imageinformation from the cameras 210 c on the augmented reality displaysubsystem 210. At block 406, the three-dimensional visualizationsub-engine 304 c may then use the user tracking information, the imageinformation, and/or any other available information to detect at leastone user interaction with the augmented reality element 600.

Referring now to FIG. 8, in this second embodiment of block 406, theuser may move their hand 702 into the three-dimensional volume 500located adjacent the physical display device 204 and perform a handgesture B that, in FIG. 8, is illustrated as a “pinch” gesture, and thatis followed by a rotational movement of the hand 702. The user trackingsub-engine 304 a may receive the user tracking information generated bythe user tracking subsystem 208 a and 208 b in response to that handmovement and provide that user tracking information to thethree-dimensional visualization sub-engine 304 c. The three-dimensionalvisualization sub-engine 304 c may also receive image informationgenerated by the camera(s) 210 c on the augmented reality displaysubsystem 210 in response to that hand movement. The three-dimensionalvisualization sub-engine 304 c may then analyze the user trackinginformation, the image information, and/or any other availableinformation to detect the hand gesture B being performed within thethree-dimensional volume 500, followed by the rotational movement of thehand 702. In the specific embodiment illustrated in FIG. 8, the handgesture B is detected adjacent the augmented reality element X-axis 700a prior to the rotational movement of the hand 702, and that handgesture B and augmented reality element X-axis 700 a may be associatedwith each other.

Furthermore, as also illustrated in FIG. 8, a user interaction may bedetected via totem device information generated in response torotational movement of the totem device 212. For example, the user mayuse their hand to rotate the totem device 212 in a plane as illustrated,and that rotational movement may be reported to the three-dimensionalvisualization sub-engine 304 c as totem device information (e.g.,reported directly by the totem device 212, detected and reported by theuser tracking sub-engine 304 a, etc.) As such, at block 406, thethree-dimensional visualization sub-engine 304 c may use the totemdevice information generated via rotational movement of the totem device212 to detect the at least one user interaction with the augmentedreality element 600.

In the second embodiment, the method 400 may then proceed to block 408where the computing system uses the augmented reality display subsystemto modify the display of the augmented reality element in the threedimensional volume adjacent the physical display device based on theuser interaction. In an embodiment of block 408, the three-dimensionalvisualization sub-engine 304 c may determine that the user interactiondetected at block 406 and, in response, modify the display of theaugmented reality element 600 in the three dimensional volume 500. Inthis example, the hand gesture A is a pinch gesture that is performedadjacent the augmented reality element X-axis 700 a, and is followed byrotational movement of the hand 702 about the augmented reality elementX-axis 700 a. Based on that detected user interaction with the augmentedreality element 600, the augmented reality visualization sub-engine 304c may modify the display of the augmented reality element 600 providedby the augmented reality display subsystem 210 to cause the augmentedreality element 600 to appear to perform a rotational movement withinthe three-dimensional volume 500 and about that augmented realityelement X-axis 700 a.

In the second embodiment, the method 400 may then proceed to block 410where the computing system uses the physical display device to modifythe display of the two dimensional element that is coordinated with theaugmented reality element. With reference to FIG. 8, the rotationalmovement of the augmented reality element 600 may cause thetwo-dimensional visualization sub-engine 304 b to modify the display ofthe two-dimensional element 602. For example, as discussed above, thetwo-dimensional element 602 may be coordinated with the augmentedreality element 600 by providing a cross-sectional view of the augmentedreality element 600, and in this example that cross-sectional view maybe defined by a horizontally oriented static plane (e.g., a plane thatincludes the augmented reality element X-axis 700 a and the augmentedreality element Y-axis 700 b in FIG. 7A, and that is parallel to thedisplay screen 204 a) that intersects the augmented reality element 600.In response to the rotational movement of the augmented reality element600, the relative orientation of the augmented reality element 600 andthat static plane may change, and result in the coordinated display ofthe two-dimensional element 704 that provides a different cross-sectionof the augmented reality element 602 as it rotates. One of skill in theart in possession of the present disclosure will recognize how across-sectional view of the augmented reality element 600 may beprovided with a vertical static plane (i.e., a static plane thatincludes the augmented reality element Y-axis 700 b and the augmentedreality element Z-axis 700 c in FIG. 7A, and that is perpendicular tothe display screen 204 a) in substantially the same manner as describedabove.

In other examples, the coordinated modification of the two-dimensionalelement 602 with the augmented reality element 600 may include providinga modified projected view of the augmented reality element 600. Forexample, the two-dimensional element 704 may provide a projected view ofthe augmented reality element 600 that illustrates different portions ofthe surface of the augmented reality element 600 as it rotates (possibleabout the augmented reality element X-axis 700 a or the augmentedreality element Y-axis 700 c rather than the augmented reality elementZ-axis 700 c in the illustrated embodiment), a projected view thatillustrates a shadow cast by the augmented reality element 602 as itrotates (e.g., in response to the augmented reality light source thatmay be positioned as desired by the user), and/or that illustrated avariety of other projected views that would be apparent to one of skillin the art in possession of the present disclosure.

In yet other examples, the coordinated modification of thetwo-dimensional element 602 with the augmented reality element 600 mayinclude a modified surface-engagement view of the augmented realityelement 600. As discussed above, augmented reality element 600 mayinclude a textured surface (not illustrated), and the two-dimensionalelement 704 may provide a surface-engagement view of the augmentedreality element 600 that illustrates how the textured surface of theaugmented reality element 602 would engage a support surface as theaugmented reality element 602 rotates, the amount of force that would beexperienced by different portions of the textured surface of theaugmented reality element 602 as it rotates when engaging a supportsurface, and/or a variety of other surface-engagement views that wouldbe apparent to one of skill in the art in possession of the presentdisclosure. While specific examples of the display of modifiedcoordination of the display of two-dimensional elements with augmentedreality elements have been described, one of skill in the art inpossession of the present disclosure will recognize that a variety oftechniques for rendering such modified coordinated two-dimensionalelements/augmented reality elements will fall within the scope of thepresent disclosure as well.

In a third embodiment, the method 400 may proceed to block 406 where thecomputing system detects a user interaction with the augmented realityelement. In the third embodiment, at block 406, the user trackingsub-engine 304 a in the augmented reality display engine 304 may receiveuser tracking information that may be generated by the user trackingsubsystem 208 a and 208 b (e.g., user tracking information related tothe location, orientation, and/or movement of any part of the user'sbody detectable by the user tracking subsystem 208 a and 208 b), and/orgenerated by any other component in the system 200. The user trackingsub-engine 304 a may then provide that user tracking information to thethree-dimensional visualization sub-engine 304 c. Furthermore, at block406 the three-dimensional visualization sub-engine 304 c in theaugmented reality display engine 304 may receive image information fromthe cameras 210 c on the augmented reality display subsystem 210.Further still, at block 406 the three-dimensional visualizationsub-engine 304 c in the augmented reality display engine 304 may receivepen device information from the pen device 214. At block 406, thethree-dimensional visualization sub-engine 304 c may then use the usertracking information, the image information, the pen device information,and/or any other available information to detect at least one userinteraction with the augmented reality element 600.

Referring now to FIG. 9, in this third embodiment of block 406, the usermay utilize the pen device 214 with the display screen 204 a of thephysical display device 204 and perform a pen device action that, inFIG. 9, is illustrated as including drawing a feature 900 on thetwo-dimensional element 602. The user tracking sub-engine 304 a mayreceive the user tracking information generated by the user trackingsubsystem 208 a and 208 b in response to that pen device action, andprovide that user tracking information to the two-dimensionalvisualization sub-engine 304 b. The two-dimensional visualizationsub-engine 304 b may also receive image information generated by thecamera(s) 210 c on the augmented reality display subsystem 210 inresponse to that pen device action, and the pen device 214 may report tothe two-dimensional visualization sub-engine 304 b (or thetwo-dimensional visualization sub-engine 304 b may detect) pen deviceinformation that describes that pen device action. The two-dimensionalvisualization sub-engine 304 b may then analyze the user trackinginformation, the image information, the pen device information, and/orany other available information to detect the pen device action beingperformed on the two-dimensional element 602. In some embodiments, thetwo-dimensional visualization engine 304 b may be configured torecognize when a user's palm is engaging the display screen 204 a alongwith the pen device 214, and operate to disregard that engagement of theuser's palm.

In the third embodiment, the method 400 may then proceed to performfunctionality that is similar to that described above for block 408,with the exception that in this embodiment the computing system uses thephysical display device to modify the display of the two-dimensionalelement displayed on the physical display device based on the userinteraction. In an embodiment of block 408, the two-dimensionalvisualization sub-engine 304 b may determine that the user interactiondetected at block 406 and, in response, modify the display of thetwo-dimensional element 602 displayed on the physical display device204. In this example, the pen device action is performed on thetwo-dimensional element 602 that is providing a cross-section of theaugmented reality element 600, and operates to add a feature 900 to (orotherwise modify) that cross-section of the augmented reality element600. In some embodiments, the two-dimensional visualization engine 304 bmay be configured to recognize when a user's palm is engaging thedisplay screen 204 a along with the pen device 214, and operate todisregard that engagement of the user's palm.

In the third embodiment, the method 400 may then proceed to performfunctionality that is similar to that described above for block 410,with the exception that the computing system modifies the display of theaugmented reality element 600 in a manner that is coordinated with thetwo-dimensional element 602. With reference to FIG. 9, the feature 900added on the two-dimensional element 602 via the pen device action maycause the three-dimensional visualization sub-engine 304 c to modify thedisplay of the augmented reality element 600. For example, as discussedabove, the two-dimensional element 602 may be coordinated with theaugmented reality element 600 by providing a cross-sectional view of theaugmented reality element 600, and in this example the feature 900 addedto that cross-sectional view may be projected onto or otherwisecoordinated with the augmented reality element 600 to provide a feature902. Furthermore, such pen device actions may be combined with therotational movement of the augmented reality element 600 such as, forexample, by using the pen device 214 to draw the feature 900 as therelative orientation of the augmented reality element 602 in thethree-dimensional volume 500, and result in the adding of the feature900 to different cross-sections of the augmented reality element 602 asit rotates (e.g., to draw a three-dimensional feature within theaugmented reality element 600.)

In a fourth embodiment, the method 400 may proceed to block 406 wherethe computing system detects a user interaction with the augmentedreality element. In the fourth embodiment, at block 406, the usertracking sub-engine 304 a in the augmented reality display engine 304may receive user tracking information that may be generated by the usertracking subsystem 208 a and 208 b (e.g., user tracking informationrelated to the location, orientation, and/or movement of any part of theuser's body detectable by the user tracking subsystem 208 a and 208 b),and/or generated by any other component in the system 200. The usertracking sub-engine 304 a may then provide that user trackinginformation to the three-dimensional visualization sub-engine 304 c.Furthermore, at block 406 the three-dimensional visualization sub-engine304 c in the augmented reality display engine 304 may receive imageinformation from the cameras 210 c on the augmented reality displaysubsystem 210. Further still, at block 406 the three-dimensionalvisualization sub-engine 304 c in the augmented reality display engine304 may receive pen device information from the pen device 214. At block406, the three-dimensional visualization sub-engine 304 c may then usethe user tracking information, the image information, the pen deviceinformation, and/or any other available information to detect at leastone user interaction with the augmented reality element 600.

Referring now to FIG. 10, in this fourth embodiment of block 406, thethree-dimensional visualization sub-engine 304 c may operate to use theaugmented reality display subsystem 210 to provide an additionalaugmented reality element 1000 for display such that it appears to belocated in the secondary zone and at least partially outside of thethree-dimensional volume 500. In the illustrated embodiment, theadditional augmented reality element 1000 is a menu-type augmentedreality element panel that may allow the user to select betweendifferent colors, textures, and/or other features that may be applied tothe augmented reality element 600. For example, such menu-type augmentedreality elements may be utilized to “drag and drop” colors, textures,and/or other features from the menu-type augmented reality element tothe augmented reality element 600 (e.g., a red color from the menu-typeaugmented reality element may be dragged through the three-dimensionalvolume 500 and dropped on the augmented reality element 600 to result inthe augmented reality element being displayed in that red color.)However, one of skill in the art in possession of the present disclosurewill recognize how additional augmented reality elements may be providedin the second zone, the tertiary zone, or other zones, in order toprovide a variety of functionality that will fall within the scope ofthe present disclosure.

As illustrated in FIG. 10, the user may utilize the pen device 214 withthe display screen 204 a of the physical display device 204 and performa pen device action that, in FIG. 10, is illustrated as including addinga feature 1002 to the two-dimensional element 602. The user trackingsub-engine 304 a may receive the user tracking information generated bythe user tracking subsystem 208 a and 208 b in response to that pendevice action, and provide that user tracking information to thetwo-dimensional visualization sub-engine 304 b. The two-dimensionalvisualization sub-engine 304 b may also receive image informationgenerated by the camera(s) 210 c on the augmented reality displaysubsystem 210 in response to that pen device action, and the pen device214 may report to the two-dimensional visualization sub-engine 304 b (orthe two-dimensional visualization sub-engine 304 b may detect) pendevice information that describes that pen device action. Thetwo-dimensional visualization sub-engine 304 b may then analyze the usertracking information, the image information, the pen device information,and/or any other available information to detect the pen device actionbeing performed on the two-dimensional element 602. In some embodiments,the two-dimensional visualization engine 304 b may be configured torecognize when a user's palm is engaging the display screen 204 a alongwith the pen device 214, and operate to disregard that engagement of theuser's palm.

In the fourth embodiment, the method 400 may then proceed to performfunctionality that is similar to that described above for block 408,with the exception that in this embodiment the computing system uses thephysical display device to modify the display of the two-dimensionalelement displayed on the physical display device based on the userinteraction. In an embodiment of block 408, the two-dimensionalvisualization sub-engine 304 b may determine that the user interactiondetected at block 406 and, in response, modify the display of thetwo-dimensional element 602 displayed on the physical display device204. In this example, the pen device action is performed on thetwo-dimensional element 602 that is providing a projection of theaugmented reality element 600, and operates to add a feature 1002 to (orotherwise modify) that projection of the augmented reality element 600.In some embodiments, the two-dimensional visualization engine 304 b maybe configured to recognize when a user's palm is engaging the displayscreen 204 a along with the pen device 214, and operate to disregardthat engagement of the user's palm.

In the fourth embodiment, the method 400 may then proceed to performfunctionality that is similar to that described above for block 410,with the exception that the computing system modifies the display of theaugmented reality element 600 in a manner that is coordinated with thetwo-dimensional element 602. With reference to FIG. 10, the feature 1002added on the two-dimensional element 602 via the pen device action maycause the three-dimensional visualization sub-engine 304 c to modify thedisplay of the augmented reality element 600. For example, as discussedabove, the two-dimensional element 602 may be coordinated with theaugmented reality element 600 by providing a projection view of theaugmented reality element 600, and in this example the feature 1002added to that projection view may be projected onto or otherwisecoordinated with the augmented reality element 600 to add an augmentedreality element feature 1004 (a three-dimensional feature in thisexample.) Furthermore, such pen device actions may be combined with thetranslational or rotational movement of the augmented reality element600 such as, for example, by using the pen device 214 as the relativeposition and/or orientation of the augmented reality element 602 withinthe three-dimensional volume 500 changes, and result in the adding ofthe feature 1004 to the augmented reality element 602 as it translatesor rotates.

Referring now to FIG. 11, in yet another embodiment, the two-dimensionalvisualization sub-engine 304 b and the three-dimensional visualizationsub-engine 304 c may operate to display the augmented reality element600 relative to other features 1100, 1102, and 1104 that are displayedvia the physical display device 204 and/or the augmented reality displaysubsystem 210 such that they appears to move relative to the augmentedreality element 600. One of skill in the art in possession of thepresent disclosure will recognize how the techniques described in theembodiments above may be utilized to display how motion of the augmentedreality element effects surfaces, cross sections, and/or other featuresof the augmented reality element 600. For example, the effects of afluid flow around the augmented reality element 600 may be visualizedvia movement of fluid by the augmented reality element 600, and detailsof the effects on that fluid flow may be displayed via coordinatedtwo-dimensional elements displayed on the display device 204.

Thus, systems and methods have been described that provide for theintegration of an augmented reality workspace with one or more physicaldisplay devices in order to provide an immersive workspace experiencethat coordinates the display of augmented reality elements with thedisplay of two-dimensional elements on a physical display device. Assuch, productivity increases in the design of products usingthree-dimensional elements displayed via the augmented reality displaysubsystem may be enhanced by providing for the display of “projections”of the three-dimensional element onto the physical display device usingthe two-dimensional element displayed thereon, allowing for thevisualization of cross-sections of the product being designed,engagement surfaces on the product being designed, and/or a variety ofother benefits that will be apparent to one of skill in the art inpossession of the present disclosure. Furthermore, the system may beenhanced using peripheral devices such a totems, which allow formanipulation of the three-dimensional element displayed via theaugmented reality display subsystem, and pens, which may be utilizedwith the physical display device to allow for the augmentation ofthree-dimensional element displayed via the augmented reality displaysubsystem.

While several features of the augmented reality workspace system havebeen described, one of skill in the art in possession of the presentdisclosure will recognize that a variety of other features may beprovided while remaining within the scope of the present disclosure. Forexample, the computing system may be configured to detect when theaugmented reality display subsystem is being utilized and, in the eventit is not, the computing system may disable the processing performed bythe three-dimensional visualization sub-engine 304 c in order to, forexample, conserve processing power (i.e., by only enabling the displayof images on the display screen 204 a of the physical display device204.) Furthermore, haptic transducers and/or other haptic devices may beprovided on the physical display device in the augmented reality displaysubsystem, and/or as part of the totem device and/or pen device, inorder to provide haptic feedback to a user that is utilizing theaugmented reality workspace system as described above.

Although illustrative embodiments have been shown and described, a widerange of modification, change and substitution is contemplated in theforegoing disclosure and in some instances, some features of theembodiments may be employed without a corresponding use of otherfeatures. Accordingly, it is appropriate that the appended claims beconstrued broadly and in a manner consistent with the scope of theembodiments disclosed herein.

What is claimed is:
 1. A physical display device integrated augmentedreality workspace system, comprising: a physical display device; anaugmented reality display subsystem; and a computing system including aprocessor that is coupled to the physical display device, the usertracking subsystem, and the augmented reality display subsystem, whereinthe computing system is configured to: define, based on physical displaydevice information associated with the physical display device, athree-dimensional volume that is immediately adjacent the physicaldisplay device and that remains stationary relative to the physicaldisplay device when the augmented reality display subsystem movesrelative to the physical display device; provide, for display on theaugmented reality display subsystem and in order to appear to be locatedin the three-dimensional volume immediately adjacent the physicaldisplay device as viewed through the augmented reality displaysubsystem, a first augmented reality element; provide, for display onthe physical display device, a two-dimensional element that iscoordinated with the first augmented reality element displayed on theaugmented reality display subsystem; identify at least one first userinteraction with the two-dimensional element displayed on the physicaldisplay device; modify, in response to the at least one first userinteraction, the display of the two-dimensional element displayed on thephysical display device; and modify, in response to the at least onefirst user interaction, the display of the first augmented realityelement by the augmented reality display subsystem in a manner that iscoordinated with the modification of the display of the two-dimensionalelement displayed on the physical display device.
 2. The physicaldisplay device integrated augmented reality workspace system of claim 1,wherein the first augmented reality element is displayed in order toappear as a three-dimensional object that is located adjacent thephysical display device as viewed through the augmented reality displaysubsystem.
 3. The physical display device integrated augmented realityworkspace system of claim 1, further comprising: a user trackingsubsystem coupled to the computing system, wherein the computing systemis configured to: receive, from the user tracking subsystem, first usertracking information; analyze the first user tracking information toidentify the at least one first user interaction with thetwo-dimensional element displayed on the physical display device.
 4. Thephysical display device integrated augmented reality workspace system ofclaim 1, wherein the two-dimensional element is coordinated with thefirst augmented reality element by displaying a cross section of thefirst augmented reality element.
 5. The physical display deviceintegrated augmented reality workspace system of claim 1, wherein thecomputing system is configured to: identify at least one second userinteraction with the two-dimensional element displayed on the physicaldisplay device; modify, in response to the at least one second userinteraction, the display of the two-dimensional element displayed on thephysical display device; and modify, in response to the at least onesecond user interaction, the display of the first augmented realityelement by the augmented reality display subsystem in a manner that iscoordinated with the modification of the display of the two-dimensionalelement displayed on the physical display device.
 6. The physicaldisplay device integrated augmented reality workspace system of claim 1,wherein the modification of the two-dimensional element includesmodifying the area occupied by the two-dimensional element displayed onthe physical display device, and wherein the modification of the firstaugmented reality element includes modifying the display of a virtualvolume of the first augmented reality element by the augmented realitydisplay subsystem.
 7. An Information Handling System (IHS), comprising:a processing system including a processor; and a memory system thatincludes system memory and that is coupled to the processing system andthat includes instructions that, when executed by the processing system,cause the processing system to provide an augmented reality displayengine that is configured to: define, based on physical display deviceinformation associated with the physical display device, athree-dimensional volume that is immediately adjacent a physical displaydevice and that remains stationary relative to the physical displaydevice when an augmented reality display subsystem moves relative to thephysical display device; provide, for display on the augmented realitydisplay subsystem and in order to appear to be located in thethree-dimensional volume immediately adjacent the physical displaydevice as viewed through the augmented reality display subsystem, afirst augmented reality element; provide, for display on a physicaldisplay device, a two-dimensional element that is coordinated with thefirst augmented reality element displayed on the augmented realitydisplay subsystem; identify at least one first user interaction with thetwo-dimensional element displayed on the physical display device;modify, in response to the at least one first user interaction, thedisplay of the two-dimensional element displayed on the physical displaydevice; and modify, in response to the at least one first userinteraction, the display of the first augmented reality element by theaugmented reality display subsystem in a manner that is coordinated withthe modification of the display of the two-dimensional element displayedon the physical display device.
 8. The IHS of claim 7, wherein the firstaugmented reality element is displayed in order to appear as athree-dimensional object that is located adjacent the physical displaydevice as viewed through the augmented reality display subsystem.
 9. TheIHS of claim 7, wherein the augmented reality display engine isconfigured to: receive, from a user tracking subsystem, first usertracking information; analyze the first user tracking information toidentify the at least one first user interaction with thetwo-dimensional element displayed on the physical display device. 10.The IHS of claim 7, wherein the two-dimensional element is coordinatedwith the first augmented reality element by displaying a cross sectionof the first augmented reality element.
 11. The IHS of claim 7, whereinthe augmented reality display engine is configured to: identify at leastone second user interaction with the two-dimensional element displayedon the physical display device; modify, in response to the at least onesecond user interaction, the display of the two-dimensional elementdisplayed on the physical display device; and modify, in response to theat least one second user interaction, the display of the first augmentedreality element by the augmented reality display subsystem in a mannerthat is coordinated with the modification of the display of thetwo-dimensional element displayed on the physical display device. 12.The IHS of claim 7, wherein the first user tracking information isreceived from a physical user interaction device that includes at leastone of a totem device and a pen device.
 13. The IHS of claim 7, whereinthe modification of the two-dimensional element includes modifying thearea occupied by the two-dimensional element displayed on the physicaldisplay device, and wherein the modification of the first augmentedreality element includes modifying the display of a virtual volume ofthe first augmented reality element by the augmented reality displaysubsystem.
 14. A method for providing an augmented reality workspacethat is integrated with a physical display device, comprising: defining,by a computing system based on physical display device informationassociated with the physical display device, a three-dimensional volumethat is immediately adjacent a physical display device and that remainsstationary relative to the physical display device when an augmentedreality display subsystem moves relative to the physical display device;providing, by the computing system for display on the augmented realitydisplay subsystem and in order to appear to be located in thethree-dimensional volume immediately adjacent the physical displaydevice as viewed through the augmented reality display subsystem, afirst augmented reality element; providing, by the computing system fordisplay on a physical display device, a two-dimensional element that iscoordinated with the first augmented reality element displayed on theaugmented reality display subsystem; identifying, by the computingsystem, at least one first user interaction with the two-dimensionalelement displayed on the physical display device; modifying, by thecomputing system in response to the at least one first user interaction,the display of the two-dimensional element displayed on the physicaldisplay device; and modifying, by the computing system in response tothe at least one first user interaction, the display of the firstaugmented reality element by the augmented reality display subsystem ina manner that is coordinated with the modification of the display of thetwo-dimensional element displayed on the physical display device. 15.The method of claim 14, wherein the first augmented reality element isdisplayed in order to appear as a three-dimensional object that islocated adjacent the physical display device as viewed through theaugmented reality display subsystem.
 16. The method of claim 14, furthercomprising: receiving, by the computing system from a user trackingsubsystem, first user tracking information; analyzing, by the computingsystem, the first user tracking information to identify the at least onefirst user interaction with the two-dimensional element displayed on thephysical display device.
 17. The method of claim 14, wherein thetwo-dimensional element is coordinated with the first augmented realityelement by displaying a cross section of the first augmented realityelement.
 18. The method of claim 14, further comprising: identifying, bythe computing system, at least one second user interaction with thetwo-dimensional element displayed on the physical display device;modifying, by the computing system in response to the at least onesecond user interaction, the display of the two-dimensional elementdisplayed on the physical display device; and modifying, by thecomputing system in response to the at least one second userinteraction, the display of the first augmented reality element by theaugmented reality display subsystem in a manner that is coordinated withthe modification of the display of the two-dimensional element displayedon the physical display device.
 19. The method of claim 14, wherein thefirst user tracking information is received from a physical userinteraction device that includes at least one of a totem device and apen device.
 20. The method of claim 14, wherein the modification of thetwo-dimensional element includes modifying the area occupied by thetwo-dimensional element displayed on the physical display device, andwherein the modification of the first augmented reality element includesmodifying the display of a virtual volume of the first augmented realityelement by the augmented reality display subsystem.